[WIP] Generic QC device and late tasks as it's first adopter#2628
[WIP] Generic QC device and late tasks as it's first adopter#2628knopers8 wants to merge 18 commits intoAliceO2Group:masterfrom
Conversation
This proposes "late tasks", which can take as input merged MOs and any QOs within the same workflow. See JIRA for more deliberations about why it's probably the best approach. Definitely not finished, I'm just saving the progress and making it available for early review.
userCodeName being the task/check/aggregator/... name set by users
Barthelemy
left a comment
Barthelemy
left a comment
There was a problem hiding this comment.
Very nice, and clear, thanks a lot !
It took me quite some time to understand (again) the Actor stuff, but I am glad I made the effort because I learned a lot.
A number of classes and struct are missing documentation (just a sentence to say what it does is enough). But it is normal at this stage.
| struct LateTaskConfig : public UserCodeConfig { | ||
|
|
||
| // fixme: we should think what should happen when a user wants a critical late task relying on a non-critical QC task. | ||
| // should it be marked as resilient? |
There was a problem hiding this comment.
this is the definition of resilient
There was a problem hiding this comment.
Indeed. The issue, which I did not explain well with the comment, is that we don't (easily) know if an upstream data processor is expendable or critical, so we don't know if we have to use "resilient" or not. However, I think we could safely use "resilient" every time a data processor is not expendable and I would try going this way.
| /// TODO: one could even consider allowing to feed late tasks with output of Reductors. | ||
| /// It could be an opportunity to refactor them as well (and rename them to Reducers, which sounds more like English). | ||
| /// TODO: to allow for more structured configuration, i see no other choice than providing the user with the late task config tree. | ||
| /// CustomParameters do not support tree-like structures. one could consider extending it, but i'm not sure if we can be fully backwards compatible. |
There was a problem hiding this comment.
I think that we do now.
|
OK, cool, thanks! Then I propose that I'll rework the commit history a bit and come with separated PRs for each change and include documentation accordingly. |
singiamtel
changed the title
2 participants
This branch demonstrates how a generic Actor could be used to offload the commonalities into one place and allow for enough customizability among different QC data processors.
There is still quite a few todos, but this already demonstrates the idea.
o2-qc-run-basic and basic.json are to play with, other jsons added in the branch will not work at the moment.